Systems and methods for presenting results of experiments

ABSTRACT

A method of automatically analyzing data from at least one data set including a plurality of process factors of interest and a process output of interest to determine the relationship between the factors of interest and the output of interest at a given significance level and preserving model hierarchy. The method includes the steps of calculating the effects of the factors of interest against the process output of interest, developing a model including the significant factors of interest and respective estimated coefficients and omitting the insignificant factors of interest, generating a representation of the model, and generating at least one graphical representation of the factors of interest.

FIELD OF THE INVENTION

The invention relates to systems and methods for assisting a user indesigning experiments. Specifically, the invention provides a wizard forguiding a user through the design of experiments.

BACKGROUND OF THE INVENTION

Design of Experiments (DOE) is used to analyze a process to determinewhich process inputs have the greatest impact on the process. Theprocess inputs, referred to as factors (e.g., temperature, quantity,etc.), have different levels. DOE allows a comparison of how differentlevels for the factors impact the process output or response. DOE usesrandomization and replication to improve the results of the experiment.A concept called blocking allows variable factors to be removed from theexperiment (e.g., differences between workers on a first shift and asecond shift). Experiments are then created using full or fractionalfactorial designs.

SUMMARY OF THE INVENTION

In one embodiment, the invention provides a method of automaticallyanalyzing data from at least one data set including a plurality ofprocess factors of interest and a process output of interest todetermine the relationship between the factors of interest and theoutput of interest at a given significance level and preserving modelhierarchy. The method uses a computer to carry out the steps ofcalculating the effects of the factors of interest against the processoutput of interest, developing a model including the significant factorsof interest and respective estimated coefficients and omitting theinsignificant factors of interest, generating a representation of themodel, and generating at least one graphical representation of thefactors of interest. The graphical representation provides a firstrepresentation corresponding to factors of interest identified assignificant and a second representation corresponding to factors ofinterest identified as insignificant. Calculating the effects includescalculating at least one of an analysis of variance test testing thesignificance of the individual effect of each factor of interest,coefficients and effect estimates for each factor of interest, and modelcomparison statistics, and identifying the factors of interest havingsignificant effects relative to the given significance level and modelhierarchy.

In another embodiment, the invention provides a method of automaticallyanalyzing data from at least one data set including a plurality ofprocess factors of interest and a process output of interest todetermine the relationship between the factors of interest and theoutput of interest at a given significance level and preserving modelhierarchy. The method uses a computer to carry out the steps ofcalculating the effects of the factors of interest against the processoutput of interest, and developing a model including the significantfactors of interest and respective estimated coefficients and omittingthe insignificant factors of interest. Calculating the effects includescalculating at least one of an analysis of variance test testing thesignificance of the individual effect of each factor of interest,coefficients and effect estimates for each factor of interest, and modelcomparison statistics, and identifying the factors of interest havingsignificant effects relative to the given significance level and modelhierarchy.

Other aspects of the invention will become apparent by consideration ofthe detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a computer system for implementing asoftware program embodying the invention.

FIG. 2 is a spreadsheet for defining a design of experiments project.

FIGS. 3A and 3B are an embodiment of the operation of a wizard fordesigning experiments.

FIG. 4 is a user interface screen displayed by the wizard.

FIG. 5 is a process specific help screen displayed by the wizard.

FIG. 6 is a process specific help screen displayed by the wizard.

FIG. 7 is a process specific help screen displayed by the wizard.

FIG. 8 is a process specific help screen displayed by the wizard.

FIG. 9 is a process specific help screen displayed by the wizard.

FIG. 10 is a process specific help screen displayed by the wizard.

FIG. 11 is a process specific help screen displayed by the wizard.

FIG. 12 is a process specific help screen displayed by the wizard.

FIG. 13 is a user interface screen displayed by the wizard.

FIG. 14 is a user interface screen displayed by the wizard.

FIG. 15 is a user interface screen displayed by the wizard.

FIG. 16 is a user interface screen displayed by the wizard.

FIG. 17 is a user interface screen displayed by the wizard.

FIG. 18 is a user interface screen displayed by the wizard.

FIG. 19 is a user interface screen displayed by the wizard.

FIG. 20 is a user interface screen displayed by the wizard.

FIG. 21 is a user interface screen displayed by the wizard.

FIG. 22 is a user interface screen displayed by the wizard.

FIG. 23 is a user interface screen displayed by the wizard.

FIG. 24 is a user interface screen displayed by the wizard.

FIG. 25 is a user interface screen displayed by the wizard.

FIG. 26 is a user interface screen displayed by the wizard.

FIG. 27 is a design of experiments summary screen generated by thewizard.

FIG. 28 is a design of experiments response entry screen generated bythe wizard.

FIG. 29 is a portion of analysis results spreadsheet generated by thesoftware program.

FIG. 30 is a portion of analysis results spreadsheet generated by thesoftware program.

FIG. 31 is a portion of analysis results spreadsheet generated by thesoftware program.

FIG. 32 is a portion of analysis results spreadsheet generated by thesoftware program.

FIG. 33 is a portion of analysis results spreadsheet generated by thesoftware program.

FIG. 34 is a portion of analysis results spreadsheet generated by thesoftware program.

DETAILED DESCRIPTION

Before any embodiments of the invention are explained in detail, it isto be understood that the invention is not limited in its application tothe details of construction and the arrangement of components set forthin the following description or illustrated in the following drawings.The invention is capable of other embodiments and of being practiced orof being carried out in various ways.

FIG. 1 illustrates a system for performing DOE according to anembodiment of the present invention. The system includes a generalpurpose computer 100. The computer 100 provides a platform for operatinga software program that guides a user through the design of experimentsand then analyzes the results of the experiment. In the systemidentified, data and program files are input to the computer 100, whichreads the files and executes the programs therein. Some of the elementsof the computer 100 include a processor 105 having an input/output (IO)section 110, a central processing unit (CPU) 115, and a memory module120. In one form, the software program for DOE is loaded into anon-transitory computer readable medium such as a memory 120 and/or aconfigured CD ROM (not shown) or other storage device (not shown). Thesoftware program includes instructions that are executed by theprocessor 105. The IO section 110 is connected to a keyboard 125 and anoptional user input device or mouse 130. The keyboard 125 and mouse 130enable the user to control the computer 100. IO section 110 is alsoconnected to a monitor 135. In operation, computer 100 generates theuser interfaces identified in FIGS. 4-34 and displays those userinterfaces on the monitor 135. The computer also includes a CD ROM drive140 and a data storage unit 145 connected to IO section 110. In someembodiments, the software program for DOE may reside on the storage unit145 or in memory unit 120 rather than being accessed through the CD ROMdrive using a CD ROM. Alternatively, CD ROM drive 140 may be replaced orsupplemented by a floppy drive unit, a tape drive unit, a flash drive,or other data storage device. The computer 100 also includes a networkinterface 150 connected to IO section 110. The network interface 150 canbe used to connect the computer 100 to a local area network (LAN), wideare network (WAN), internet based portal, or other network 155. Anysuitable interface can suffice, including both wired and wirelessinterfaces. Thus, the software may be accessed and run locally as fromCD ROM drive 140, data storage device 145, or memory 120, or may beremotely accessed through network interface 150. In the networkedembodiment, the software may be stored remote from the computer 100 on aserver or other appropriate hardware platform or storage device.

In one embodiment, the software is an add-in running in Microsoft®Excel®. A user loads the software onto the computer 100, and when theuser starts up Excel® a menu selection for the add-in appears on a menubar. By clicking through the menu selection and any submenus, the useris provided with a DOE menu. In some embodiments, the DOE menu providesthe user with four choices: a DOE planning worksheet, a design wizard, arun default analysis, and a run custom analysis. Clicking the DOEplanning worksheet opens a new worksheet 200 (FIG. 2). The worksheet 200includes a plurality of cells for defining experiments. The worksheet200, while optional, assists a user in planning experiments by havingthe user provide all the information that will be needed to design theexperiments. In addition, spaces are provided for information that isuseful for implementing the experiments (e.g., the process owner's name,the objective of the experiments, etc.).

Once the spreadsheet 200 has been completed, the user selects the designwizard function. FIGS. 3A and 3B show the operation of an embodiment ofa DOE wizard. As shown in FIG. 4, the wizard provides a plurality ofnavigation buttons 405-430 and a progress bar 435 for each screen.Clicking an exit button 405 exits out of the wizard, deleting allpreviously entered data. Clicking a help button 410 opens a processspecific help window with instructions for the particular portion of theDOE presently displayed by the wizard (FIGS. 5-12). Clicking a resetbutton 415 takes the user back to the start of the wizard, erasing allpreviously entered data. Clicking a back button 420 takes the user onescreen back in the wizard. Clicking a next button 425 moves the user tothe next screen. Clicking a finish button 430 causes the software todesign the experiments based on the data entered into the wizard. Theback button 420 is not available on a first screen 440. The next button425 is only available when all necessary data has been entered for aparticular screen of the wizard (the wizard provides default values incertain instances that do not need to be modified and some of therequested information is optional and need not be entered). The finishbutton 430 is only available when on the final screen of the wizardafter all the necessary data has been entered. The progress bar 435provides an indication of how far the user has progressed through thewizard.

The wizard provides two modes: (1) question and answer mode or (2) DOEmap mode (see screen 440). The question and answer mode provides a highlevel of guidance to the user, asking questions for each step of design.The DOE map allows a user with more experience to select the type ofexperiments directly.

Referring back to FIG. 3A, if the user selects the question and answermode (step 450), the wizard asks the user for the number of levels foreach factor (step 455). The selections include only two levels for eachfactor or at least one factor having more than two levels (FIG. 13). Ifthe user selects the DOE map mode (step 450), the user is presented witha map (FIG. 14) showing the available experiments and the criteria foreach. The user selects a design of experiments from the map (step 460).

If the user selects the only two levels per factor option in thequestion and answer mode (step 455) or selects the full factorial, highresolution fractional factorial, or low resolution fractional factorialexperiments in the DOE map mode (step 465), the wizard continues withrequesting the user to enter the number of factors to be used in theexperiments (step 470) (FIG. 15 for the question and answer mode andFIG. 16 for the DOE map mode). If the user selected five or more factors(step 472) the wizard prompts the user to select whether the experimentsare for screening or testing ruggedness (step 473) (if the user selectsfour or less factors, this selection input is not provided). Screeningis an economical experiment designed to examine a large number ofpossible factors to determine which of the factors might have thegreatest effect on the outcome of the test (generally using highresolution fractional factorial design). Testing for ruggedness is aneconomical experiment designed to establish the ruggedness of a processto a large number of factors. This often involves the destructivetesting of parts and typically only evaluates possible main effects(generally using low resolution fractional factorial design).

Next the user is presented with a screen having boxes for entering aname, a type, a unit type, a first level, and a second level for each ofthe factors chosen at step 470 (step 475) (FIG. 17A). The user is ableto select the type of each factor as “categorical” or “numerical.” Acategorical factor has a discrete number of values based on categoriesor groups. For example, a factor for location could be limited to eastand west. A numerical factor has the possibility of a range of numericvalues. For example, a factor for temperature theoretically has aninfinite number of values within a range.

Next, the user is presented with a grid (FIG. 18). In the question andanswer mode, the grid allows the user to select the number of runs to beexecuted for the experiments (step 480). This essentially selects thedesign of experiments from the full factorial, high resolutionfractional factorial, or low resolution fractional factorial experimentdesigns. The grid allows the user to select the number of runs for thepreviously entered number of factors. For example, in FIG. 18, the userhad selected five factors, and is able to choose runs of 8, 16, or 32(highlighted in the rectangle). In the DOE map mode, the user previouslyselected the type of experiments. Therefore, only one selection isavailable in the grid. When the type of experiments is selected, averification 482 of the type of experiments is shown below the grid(FIG. 19). In some embodiments, the wizard allows only a low resolutionfractional factorial design when the user selects ruggedness testing. Insome embodiments, the wizard only allows a high resolution fractionalfactorial design when the user selects screening.

Next, if available for the particular design chosen, the user isprompted to select whether to use blocks for the test or not (step 485)(FIG. 20). If the user elects to use blocks, the wizard prompts the userto enter the number of blocks to use, the blocking factor name, and thename of each block (step 490) (FIG. 21).

In the next screen (FIG. 22), the user is provided with data about thechosen experiments including the power 500. The wizard also shows howmany replicates are required to raise the power to 80% 505 and 90% 510.The wizard prompts the user to enter the number of replicates the userdesires (step 515). If at step 485, the user had selected not to enterblocks, the wizard moves directly to the replicates screen (FIG. 22)(step 520). If the user selects two or more replicates after having notselected blocks (step 525), the wizard asks whether the user wishes torun the replicates in blocks (step 530) (FIG. 23).

On the next screen (FIG. 24), the user is prompted to select the numberof responses for the experiments and is then able to enter a name andtype of units for each response (step 535). Finally, the user isprompted to enter a significance level (step 540). Once the significancelevel is selected, the finish button is highlighted and the user clickson the finish button to design the experiments (step 545).

If in the question and answer mode at step 455 the user selected morethan two issues or in the DOE map mode the user selected full factorial,the wizard prompts the user to enter the number of factors (step 550).If the user selects five or more factors (step 552) the wizard promptsthe user to select whether the experiments are for screening or testingruggedness (step 553) (if the user selects four or less factors, theselection input is not provided).

Next the wizard prompts the user for the number of levels for eachfactor (step 555), and the factor and level information (step 560) (FIG.25). Where three or more levels are used, unlike the factor informationfor factors with only two levels, the wizard limits the type of factorto categorical. A confirmatory notification 562 indicates this to theuser (see FIG. 17B).

Next, in FIG. 26, the wizard prompts the user for the number ofreplicates (step 565), and if the replicates are greater than one,allows the user to select blocking (step 570). The user is then promptedto select the number of responses for the experiments and is then ableto enter a name and type of units for each response (step 575). Finally,the user is prompted to enter a significance level (step 540). Once thesignificance level is selected, the finish button is highlighted and theuser clicks on the finish button to design the experiments (step 545).

FIGS. 27 and 28 are a sample of a design of experiments. The experimentshave three factors, using full factorial experiments with onereplication and one response. The experiments were run and response dataentered into the table. Once all the experiments have been run, and theresults entered into the table, the user selects a run default analysisoption from a pull-down menu. The system runs the analysis and providesthe results of the analysis in several forms shown in a responseworksheet (FIGS. 29-34).

The results are given in an effect table 600, an ANOVA table 605, arecommended model 610, a half normal plot 615, a Pareto chart 620, anormal probability plot 625, a versus fits plot 630, a versus order plot635, a histogram 640, a plurality of main effect plots 645-655, aplurality of interaction plots 660-665, and a cube plot 670. For themain effect plots 645-655, the interaction plots 660-665, and the cubeplot 670, a pull-down menu 675 allows the user to determine which factoror interaction to show in each graph.

The software analyzes the data, and determines which factors andinteractions are significant. The software then highlights thesignificant factors and interactions and produces a recommended model.For example, the half normal plot highlights (e.g., by color and symbol)the factors or interactions that are significant. In the example shown,factors A and B and interaction AB are significant and are shown asgolden squares. The other factors and interactions are shown as bluediamonds. Each of the factors and interactions are also labeled in thechart. Thus a user can quickly identify the significant factors bysimply viewing the plot.

Similarly, in the Pareto chart, the factors and interactions that aresignificant are grouped together and shown in gold, while the lesssignificant factors and interactions are shown in blue.

A recommended model (in the example: Response1=0.7*A+1.825*B+2.2*AB+13.825) is automatically generated by thesoftware, providing a user who is not fluent in the analysis with anoptimum model or saving a fluent user the time needed to generate themodel.

Various features and advantages of the invention are set forth in thefollowing claims.

What is claimed is:
 1. A method of automatically analyzing experimentaldata from at least one data set including a plurality of process factorsof interest and a process output of interest to model the relationshipbetween the factors of interest and the output of interest at a givensignificance level and preserving model hierarchy such that the modelincludes the factors of interest involved in a significant interaction,the method comprising: calculating, using a computer, effects of thefactors of interest on the process output of interest, includingperforming an analysis of variance test to calculate a significance ofthe individual effect of each factor of interest and interactionsbetween two or more of the factors of interest, calculating coefficientsand effect estimates for each factor of interest, and calculating modelcomparison statistics, comparing the significance calculated for eachfactor of interest and each interaction to the given significance leveland identifying a factor of interest or an interaction as having asignificant effect if the significance calculated for the factor ofinterest or the interaction satisfies the given significance level andidentifying a factor of interest or an interaction as having aninsignificant effect if the significance calculated for the factor ofinterest or the interaction does not satisfy the given significancelevel; developing, using the computer, a recommended model includingeach factor of interest and interaction identified as having asignificant effect and each factor of interest involved in aninteraction identified as having a significant effect multiplied by thecoefficient for the factor of interest and omitting each factor ofinterest and interaction identified as having an insignificant effect,the recommended model representing the relationship between significantfactors of interest and the process output of interest; generating arepresentation of the model; and generating at least one graphicalrepresentation of the factors of interest, the graphical representationproviding a first representation corresponding to the factors ofinterest and interactions identified as having a significant effect,including factors of interest involved in a significant interaction, anda second representation corresponding to the factors of interest andinteractions identified as having an insignificant effect.
 2. The methodof claim 1, further comprising automatically determining therelationship between the factors of interest and a second output ofinterest at a given significance level and preserving model hierarchy.3. The method of claim 1, wherein the at least one graphicalrepresentation of the factors of interest includes a half normal graph.4. The method of claim 1, wherein the at least one graphicalrepresentation of the factors of interest includes a Pareto chart. 5.The method of claim 1, further comprising displaying, at least onegraphical representation of the factors of interest, the factors ofinterest identified as having a significant effect in a first color anddisplaying the factors of interest identified as having an insignificanteffect in a second color, the first color different from the secondcolor.
 6. The method of claim 1, further comprising displaying, at leastone graphical representation of the factors of interest, the factors ofinterest identified as having a significant effect in a first shape anddisplaying the factors of interest identified as having an insignificanteffect in a second shape, the first shape different from the secondshape.
 7. The method of claim 1, further comprising displaying, at leastone graphical representation of the factors of interest, the factors ofinterest identified as having a significant effect in a first color anda first shape and displaying the factors of interest identified ashaving an insignificant effect in a second color and a second shape, thefirst color different from the second color and the first shapedifferent from the second shape.
 8. The method of claim 1, furthercomprising grouping the factors of interest identified as having asignificant effect together on the at least one graphical representationof the factors of interest.
 9. The method of claim 1, wherein the methodis performed by an add-in of spreadsheet program.
 10. The method ofclaim 1, wherein the model and the at least one graphical representationof the factors of interest are displayed on a spreadsheet. 11.Non-transitory computer-readable medium encoded with a plurality ofprocessor-executable instructions for automatically analyzingexperimental data from at least one data set including a plurality ofprocess factors of interest and a process output of interest to modelthe relationship between the factors of interest and the output ofinterest at a given significance level and preserving model hierarchysuch that the model includes the factors of interest involved in asignificant interaction, the instructions comprising: calculatingeffects of the factors of interest on the process output of interest,including performing an analysis of variance test to calculate asignificance of the individual effect of each factor of interest andinteractions between two or more of the factors of interest, calculatingcoefficients and effect estimates for each factor of interest, andcalculating model comparison statistics; comparing the significancecalculated for each factor of interest and each interaction to the givensignificance level and identify a factor of interest or an interactionas having a significant effect if the significance calculated for thefactor of interest or the interaction satisfies the given significancelevel and identify a factor of interest or an interaction as having aninsignificant effect if the significance calculated for the factor ofinterest or the interaction does not satisfy the given significancelevel; developing a recommended model including each factor of interestand interaction identified as having a significant effect and eachfactor of interest involved in an interaction identified as having asignificant effect multiplied by the coefficient for the factor ofinterest and omitting each factor of interest and interaction identifiedas having an insignificant effect, the recommended model representingthe relationship between significant factors of interest and the processoutput of interest; generating a representation of the model; andgenerating at least one graphical representation of the factors ofinterest, the graphical representation providing a first representationcorresponding to the factors of interest and interactions identified ashaving a significant effect, including factors of interest involved in asignificant interaction, and a second representation corresponding tothe factors of interest and interactions identified as having aninsignificant effect.
 12. The computer-readable medium of claim 11,further comprising instructions for automatically determining therelationship between the factors of interest and a second output ofinterest at a given significance level and preserving model hierarchy.13. The computer-readable medium of claim 12, further comprisinginstructions for developing a second model for the second output ofinterest including the significant factors of interest and respectiveestimated coefficients and omitting the insignificant factors ofinterest.
 14. The computer-readable medium of claim 12, furthercomprising instructions for displaying the model in a first worksheetand displaying the second model in a second worksheet.